Motion video consists of a sequence of image frames. Motion estimation algorithms exploit the act that these frames do not change significantly in time. Motion vector fields are calculated that describe the displacement of all the pixels in a frame to an earlier frame. The image is usually divided into a grid so that a single motion vector is associated with a group (block) of pixels to reduce computational complexity.
Motion estimation can be performed using a number of different methods, including, for example:                Block-matching techniques as described in section II of De Haan, G.; Biezen, P. W. A. C. “An efficient true-motion estimator using candidate vectors from a parametric motion model,” Circuits and Systems for Video Technology, IEEE Transactions on Vol. 8, Issue 1, February 1998, pp. 85-91.        Gradient based techniques as described in Horn, B. K. P. & B. G. Schunck, “Determining Optical Flow,” Artificial Intelligence, Vol. 16, No. 1-3, August 1981, pp. 186-203.        
FIG. 1 illustrates the known block matching technique between a sequence of two images and demonstrates that block-matching motion estimators typically attempt to find a block of pixels (30) in an earlier frame that has a high correlation to an equally sized block of pixels (20) in the current frame. The correlation is usually measured by a form of error function such as the sum of absolute differences (SAD) or the mean squared error (MSE). The displacement between the best matching blocks represents the motion vector (40). ‘Full search’ motion estimators choose to search for a matching block across the full range of the previous image. However, to reduce computation, most motion estimators reduce the search window for matching blocks by defining a maximum search area (10) and either perform a full-search within the window or evaluate only a number of ‘candidate’ blocks. This search area effectively limits the maximum motion vector size that is supported, and hence the fastest motion that can successfully be estimated as shown in FIG. 2.
Motion estimation is useful for several applications including, for example, motion compensated interpolation, where a number of interpolated frames are reconstructed and displayed between original images in a sequence. This reduces the motion judder of moving objects in the image sequence. The motion vectors here can be defined at any point in time between the two images, for example the midpoint, and are also used to reconstruct the interpolated frame using motion compensation.
A further application is video compression, for example, where spatial and/or temporal redundancy is removed. Motion vectors are again used to describe the displacement of image pixels between successive frames and, in this application, are transmitted in place of image data, thereby reducing the amount of data needed to represent the image sequence.
Other applications that use motion estimation to enhance video sequences include de-interlacing and noise reduction.
Detection and measurement of motion blur is also relevant to the derivation of motion vectors. Motion blur is an effect that occurs in natural image sequences as cameras attempt to capture moving objects. If an object moves while the camera shutter is open, the film acts as an integrating medium and the motion that has occurred is visible as a blur along the trajectory of motion. This known effect is shown in FIG. 3 and is effectively a low pass filtering operation on the moving image pixels.
The faster an object is moving relative to the camera, the greater the motion blur. Since motion blur occurs only along the trajectory, detail present on the moving objects is low-pass filtered in this direction only. This can be seen further in FIG. 3 where detail of the front and rear edges of moving vehicles are not visible, whereas the roof and base edges can still be distinguished. The level of sharpness of edges in an area of the image can be used as an indication of the level of motion. If sharp edges are detected in a particular direction, it can be deduced that in a perpendicular direction, fast motion is unlikely.
In order to successfully estimate motion in a wide variety of image sequences, large vector ranges are required. This increases the search area, hence the number of pixels that are required to be available during the calculation and evaluation of motion vectors within this region.
In order to perform motion estimation with large search areas, the transfer of large amounts of pixel information between processing blocks is necessary which demands large bandwidth.
Pixel storage, when incorporated into a hardware implementation, is expensive and bulky therefore it is desirable to reduce the number of available pixels required in the motion estimation and compensation stages.
Some schemes have previously been proposed to reduce the cost of implementing a large search area and these techniques will be known to those skilled in the art. For example U.S. Pat. No. 6,687,303 describes a scheme where neighbouring pixels in blocks to be evaluated are sub-sampled by 4:1 or 2:1. A similar scheme is described in U.S. Pat. No. 6,317,136 and also U.S. Pat. No. 5,982,910 in which 2:1 sub-sampling is performed on incoming pixel data. Simple sub-sampling such as this is successful in reducing complexity, however useful image information is discarded in the process. This can lead to a reduction in the quality of motion estimation as details in the image sequence, that aid the estimation and convergence of vectors, may be lost.
We have appreciated that as the speed of motion of an object in an image increases, the degree of low-pass filtering on the object in the image increases due to the motion blur and less detail is visible along the motion trajectory. Therefore, the faster the motion, the less high frequencies are present in the image at this location. It is known from the sampling theorem that the higher the frequency content in a signal, the higher the sampling rate must be in order to fully reconstruct the signal. It therefore follows that as the speed of motion increases, the lower the required sampling rate becomes.
As an extension of this, objects that are moving in a horizontal direction become blurred along the horizontal only and, similarly, vertically moving objects lose detail in this direction but retain horizontal detail. Therefore, for pixels undergoing motion in a certain direction, there will be relatively low frequency content along this trajectory, while frequency content perpendicular to this direction is maintained. This implies that the sampling frequency required along the trajectory of motion is lower than that required in the perpendicular direction. Therefore, in order to minimise the loss of important image details, pixels undergoing horizontal motion can be sub-sampled horizontally and pixels undergoing vertical motion can be sub-sampled vertically.
We have appreciated that in order to retain maximum detail in the search area, all pixels should be maintained. However, pixel storage is expensive and the transfer of large amounts of pixel information requires large bandwidth. Images which include objects in motion include blurring and we have appreciated that in such cases the sampling frequency along the direction of motion can be decreased without experiencing a significant loss of image detail.
Preferred embodiments of the invention perform a sub-sampling of the pixels within a search area in a predefined pattern. The pattern of sub sampled pixels varies throughout the search area in order to reduce the number of pixels that are stored, and, thus, to reduce the processing and memory requirements of the system, while maintaining important image details.
Preferred embodiments of the invention increase the degree of sub-sampling of image pixels as the distance from the centre of the search area increases since pixels of the block which appear furthest from the centre are due to the fastest motion and can be sub sampled without a loss in image detail.
Preferred embodiments of the invention maintain high pixel resolution close to the centre of the search area in order to maintain detail of static or slow moving blocks. The size of this high resolution area may be influenced by the maximum available search-area memory.
The invention in its various aspects will now be defined in the claims, to which reference should now be made.